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Chapter 1: Introduction 



A Little Book of Evaluation has been written specifically to help staff in Connexions services and partner organisations. It aims to encourage 
practitioners to reflect on their practice within the context of developing a Connexions service, and to examine ways in which a clear evaluation 
strategy can be undertaken in order to help determine what works and what doesn't. 

The purpose of this booklet is to provide a useful overview of the issues and a robust framework, to enable project managers and practitioners 
to evaluate and assess their work effectively. It provides practical guidance to encourage reflective practice in terms of 'what works and what 
doesn't', and to inform the 'where am I now' and 'what do I want to achieve' type of questions. It does not seek to be a definitive resource on 
research methodology; rather, it is expected to provide pointers that can be cross-referenced and expanded by referring to the Connexions 
Service Performance Management Toolkit, produced with the help of GHK Ltd, in association with the Department for Education and Skills. 

1.1 Evidence- Based Practice and the Development of the Connexions Service. 

Before discussing the nature and role of evaluation, it seems relevant to look at the emergence of the concept of evidence-based practice. Within 
this concept, evaluation is seen as a way of investigating 'what works best', with the purpose of basing future policy and practice upon the results 
of the investigation. 

David Blunkett (DfEE, 2000a) emphasised the need for government policies to be underpinned by good research. Within the Connexions Service, 
the term 'evidence-based practice' implies that innovations need to be piloted and evaluated. This should help to identify areas of good and 
illuminating practice that can inform the implementation and development of policies and practices at local, regional and national levels. 

The concept of evidence-based practice originated in medicine, and has now been translated to a number of fields of professional practice, 
including mental health, education and social work. Proponents of evidence-based practice contend that it has the potential to: 

• help ensure that new initiatives are likely to be successful, as they have been proven to work in a similar context; 

• maximise efficiency, especially in the light of scarce resources; 

• give value for money research that will directly link to practice; 

• support the transparency and accountability of decision-making; 

• empower practitioners and encourage self-directed learning for staff; 

• enhance multi-disciplinarity (Trinder, 2000). 
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The Connexions Service: drawing upon and promoting evidence-based practice 

ne of the eight key principles of the Connexions Service is using 'Evidence-Based Practice'. This key principle highlights the important role of 
research and evaluation in the development of an evidence-based culture. 



r 



Illustrative case study to demonstrate 'evidence- based practice' 



One of the Pilot Connexions projects undertaken in Lincolnshire was to develop and increase understanding of young people's 
information needs in respect of content, presentation and availability. 

As part of the process, nine consultative focus groups were held with a range of young people. The findings were used to develop an 
understanding of their needs. These identified needs were then 'matched' to existing provision. 

The evidence obtained from the exercise highlighted information shortfalls in key areas such as housing, handling finances, the law and 
state benefits. At the same time, there was over- provision of information about health, sex and drugs. Young people were not impressed 
with lengthy typed text and preferred humour and graphics. They also preferred fewer items of information on the same topic, and were 
seeking more in the way of signposting, preferably from a 'one-stop' approach. 

These findings enabled the pilot project to be very clear about what would work and what would not. They highlighted that much existing 
effort by a number of agencies was not meeting the needs of young people. With the support of these agencies, a simple 'one-stop' 
approach to written information has been developed in a format that will be attractive to the target audience. As an outcome of the 
evaluative process, 53,000 copies of the 'Ultimate Survival Guide' will be distributed throughout the county both to individuals and 
through appropriate access points. 
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1.2 What is Evaluation? 

In its simplest form, evaluation seeks to identify good and interesting practice, and based upon that, use the findings to inform future policy and 
practices. It is important to recognise that learning from one's failures, as well as the things that were a great success, can provide valuable 
insights into what works and what doesn't. 

Evaluation is a process used to determine what has happened during a given activity, whether a programme or activity is working, and whether 
the initial intent has been carried out and achieved. 

Evaluation is more than testing or measuring; it includes asking and answering basic questions about efforts and results, and setting the stage 
for a culture of evidence-based practice that encourages reflection, collaborative partnerships and professional development. 

Evaluation is not an end product but a process expected to take place across the phases of a project or activity: from the initial planning and 
forecasting, to the implementing, delivery, and follow-up stages. 

Effective evaluation requires: 

• time; 

• energy; 

• commitment; 

• the ability to share effective strategies and practices (e.g. review, monitoring) and adopt those that seem to be applicable in a given 
situation. 

Another important role for evaluation is to contribute to the critical appraisal of evidence-based practice. Specifically, within the Connexions 
Service, a number of strategies and initiatives that are informed by research are implemented. Thus, evaluation of these initiatives and 
programmes can function not only as a means of accumulating evidence to inform future developments, but also as a critical appraisal of the 
evidence-based practice itself. 

Connexions services will be required to evaluate their performance, both in relation to the targets set for them, and through more detailed 
monitoring of performance and client satisfaction surveys. 
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1.3 Monitoring and Evaluation 

It is important to differentiate between project monitoring and project evaluation: 

• Project monitoring is putting in place targets and milestones to measure progress and achievement. 

• Project evaluation is putting in place mechanisms to assess the effectiveness or otherwise of the project and to reflect on the lessons 
learned. 

More specifically, evaluation should address key aspects of a project from its initial proposal to ensuring that the objectives and deadlines are 
met and appropriate dissemination is achieved. It is important that every project is assessed in terms of: 

• objectives; 

• roles of individuals/groups involved; 

• task - design of research activities, piloting and implementation; 

• outcomes - deliverable products/ systems and the potential impact of the findings. 

A G antt Chart can be a useful tool when scheduling and monitoring progress, and an example is given on pages 30 and 31. In addition, a 
downloadable sample can be obtained at the Centre for G uidance Studies website http://www.derby.ac.uk/ceqs 

1.4 A Pragmatic Approach to Evaluation 

Practitioners have all sorts of competing pressures on their time and workload. With this is mind, evaluation could be viewed either as an extra 
burden, or as something that is an integral part of a practitioner's work, i.e. a regular part of one's professional practice and an aid to personal 
development. 

Nowadays most individual project managers, practitioners and organisations have to provide evidence-based information relating to the 
effectiveness of their work, e.g. process management, and achievement of targets. Project managers are not full-time researchers, but are often 
involved in action research which focuses on an immediate application of the results rather than on the development of theory or more general 
applications. Their aim is to improve practice and to support others in improving practice through reflection and continuous professional 
development. 
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Although this process can seem daunting initially, there are some easy approaches that can be followed in developing and implementing an 
evaluation strategy, supported by tried and tested methods and procedures. 

A number of evaluation approaches can and should be taken, as long as they are relevant to the scope and depth of the project design. They 
should provide an appropriate level of synergy relating to the project objectives. For project managers and practitioners in particular, it is 
important to adopt a pragmatic and informed approach towards evaluation. This will bring rewards, such as having evidence to demonstrate 
the benefits of investing in new and exciting initiatives, as well as continuing with traditional approaches that have a proven track record. 



Illustrative case study - A pragmatic approach 

A Connexions Pilot project that investigated methods of consulting with young people and parents, developed a number of approaches 
to the task and then trialled and evaluated the effectiveness of each method in relation to the specific audience. 

The evidence from this first stage enabled the various approaches to be refined, and suggested the most appropriate method to be used 
in a wider consultation process. Findings supported the use of some 'tried and tested' techniques such as focus groups and confirmed 
that there was a general level of indifference to questionnaires. 

Where new approaches were tested - such as interactive software, - they proved more popular with some groups than others, again 
underlining the importance of developing a battery of consultative systems to suit the audience. 

By adopting the trial, test, develop and evaluate approach, this project was able to have confidence in the eventual findings from the 
consultations. 
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Chapter 2: Getting Started 



This chapter provides a brief guide to the issues that should be considered before embarking on an evaluation. After breaking down the five key 
elements of an evaluation framework and the issues relating to each stage, it moves on to consider the ethics and values of research and 
concepts of reliability and validity. 

2.1 An Evaluative Framework 

There are six main components of an evaluative framework. 

• Key feature: the broad aim and objectives of the project. 

• Rationale: the underlying reasons for what you want to achieve and why. 

• Process: how you are going to approach the evaluation and how you will go about it. 

• Inputs: the resources needed to achieve your aims and objectives. 

• Indicators of success: the measure of whether the purpose has been achieved. 

• Reporting and dissemination strategy: the methods used to present and promote the outcomes and findings. 
2.1.1 Key Feature 

It is important to be clear what the broad aim of the evaluation is. For example, a broad aim of a Connexions Partnership might be: 



To increase the effectiveness of Careers Education and Guidance for all throughout the partnership area. 
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nee the broad aim of the project has been established, it is necessary to unpack or 'operationalise' the different concepts and issues contained 
within it in order to formulate the objectives. The key concepts contained within the example on page 10 are highlighted in bold. In order to 
ensure that the aim is achieved, research objectives for each must be defined. For example: 



' • To examine the extent of current careers education and guidance provision in the partnership area. ^ 

• To compare and contrast local provision with careers education and guidance provision in other areas. 

• To identify which aspects of current careers education and guidance that clients find valuable, and whether there are any gaps in 
provision or demand. 

• To measure the effectiveness of current careers education and guidance provision and identify how this can be improved. 



2.1.2 Rationale 



The rationale should be brief and to the point, encapsulating the reasons why there is a need to undertake the evaluation and the justification 
for so doing. For example, the rationale for the study on page 9 could be: 



Effective and appropriate careers education and guidance will ensure that all young people are equipped to make rational and relevant 
decisions about their future education, training and employment. 



2.1.3 Process (methodology) 

The evaluation process, or methodology, encompasses both the theoretical approach, and research techniques used for data collection. 
Developing the evaluation process can be a difficult task, but being clear about what you are trying to achieve will make it less complicated or 
confusing. 
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A number of factors should be considered when preparing to evaluate a particular activity. 

• What structures are in place to help you collect, analyse and interpret data, and to inform decision-making? Identifying these 
structures will help you to recognise the practical and logistical constraints that limit any evaluation. 

• What is/are the most appropriate method(s) of data collection - qualitative and/or quantitative. The main distinction between the 
two is: 

quantitative research methods deal with numbers and usually employ statistical techniques; 

qualitative research methods typically deal with verbal descriptions that are recorded and later transcribed for analysis and 
interpretation. 

2.1.4 Inputs 

Evaluation can be both resource and labour intensive. It is therefore important to be clear a bout the availability, roles, responsibilities and agreed 
inputs, financial and otherwise, of key stakeholders, who can include funders, partners, practitioners and clients. Some thought must also be 
given as to how their needs in relation to the project are to be established and met. 

2.1.5 Indicators of success 

In order to monitor the progress of the evaluation and ultimately assess whether it achieved its aims and objectives, it is necessary to develop a 
number of 'indicators of success' or 'points of reference' to help to measure the distance travelled and key milestones achieved. To be effective 
these indicators must be linked to, and derived from, the key feature, rationale and process. 
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2.1.6 Reporting and dissemination strategy 

It is important to have an effective reporting and dissemination strategy in place before you begin the evaluation because the way in which you 
intend to use, report and share your findings will have implications for how you gather and analyse the information. The final chapter deals with 
these issues in more depth. Below are some of the questions you should consider when formulating your strategy. 

• Will the research findings be predominantly quantitative or qualitative? 

• Who is/are your audience(s)? 

• Who will use the findings and for what will they use them? 

• What is the clearest way to present the information and is more than one format required? 

• Who else will find the results useful? 

• What is the most effective way to make the results available to them and others? 
2.2 Ethics and Values 

Before embarking on an evaluation project, it is important to consider the simple rules and protocols that should be observed. 

1 Informed consent: Take care to ensure that the relevant persons and authorities have been consulted and that the necessary 
permission and approval have been obtained. 

2 Obtain explicit authorisation: Ensure you have permission before you observe, examine files, or use quotations, correspondence 
or other documentation. 

3 Negotiate with those affected: Your work should take into consideration the responsibilities and wishes of others; not everyone will 
want to be directly involved in evaluation activities. 

4 Involve participants: Encourage the study's participants to express their views at different stages of the evaluation, not just in the 
data-collection phase. Failure to involve individuals appropriately can effect the reliability and validity of the project (see Section 2.3). 

5 Negotiate descriptions of people's work: Allow those being described and/or discussed to challenge your accounts on the 
grounds of fairness, relevance and accuracy 
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6 Report the progress: Keep the work visible and remain open to suggestions so that foreseen and unforeseen consequences can be 
taken into account. 

7 Consider the audience(s): It may be necessary to produce reports for a variety of audiences each operating at a different level and 
requiring a different focus. If a number of different reports are required, this should be negotiated at the outset. 

8 Confidentiality and anonymity: The project manager must accept responsibility for assuring and maintaining confidentiality and 
anonymity. Individuals or organisations that wish to remain anonymous in your evaluation activities, should not be named, and steps 
should be taken to ensure that they are not traceable. 

2.3 Reliability, Validity and Accuracy 

Reliability and validity are essential to the effectiveness of data collection, and to the analysis and interpretation of the results. ne should, 
therefore, always attempt to improve the reliability and validity of research approaches. However, precise determination of the degree to which 
they are achieved can be elusive at times, particularly in the case of validity. 

Reliability refers to the degree of consistency of the assessment procedure or instrument. Whatever it is measuring, it does so consistently across 
settings and over time. 

Validity refers to the quality of the data-gathering instrument and whether meaningful issues, themes and conclusions can be extracted from the 
resulting information. In order to assess validity, the researcher must question whether the instrument, or procedure, measures what it is designed 
to measure, and in so doing meets the aims and objectives of the study. 

It is somewhat difficult to determine the reliability and validity of some data-gathering instruments or procedures, particularly observation, 
unstructured interviews, or questionnaires with open questions that produce responses that are not easily quantifiable. In these instances, it is 
important to ensure that the information provided by the respondent is accurate: that is, the data is collected without making mistakes and the 
response is a true reflection of events at the time it was collected. Simple logic checks can often be used to verify the accuracy of the information 
provided. 



A Little Book of Evaluation 



Chapter 3: The Research Process 



There are a number of different methods that can be used when undertaking an evaluation. In this section, examples of a range of methods that 
are commonly used are presented. A very brief summary of key issues is also provided to enable you to reflect on the advantages and 
disadvantages of each approach. 

3.1 Sampling 

The primary purpose of evaluation is to discover principles and trends that may have a universal application. However, to study the entire 
population and then arrive at generalisations would be impracticable - if not impossible. Fortunately, the process of sampling makes it possible 
to draw valid inferences or generalisations through consultation with a relatively small proportion of the population in question. 

• A population is any group of individuals and/or cases that have one or more characteristics in common that are of interest to the 
evaluator. For example, the Connexions Service has been designed to support young people aged between 13 and 19. Therefore, 
13-19 year olds constitute the whole population of Connexions- related research. 

• A sample is a small proportion of a population selected for observation and analysis. For example, in order to make inferences about 
factors such as disaffection, poor attendance, lack of motivation etc. for this population as a whole, one could select a sample from 
the population of 13-19 year olds within a school. 

Samples can be selected in a number of different ways. These include: 

• simple random sampling - where a percentage of individuals is selected completely at random by 'picking names out of a haf ; 

• systematic random sampling - where a random start point is selected and then every 'nth' person from that point onwards; 

• stratified random sampling - where individuals are grouped according to certain criteria or 'strata', e.g. gender; the sample is then 
selected to reflect the actual profile of the population in terms of the specified criteria, in this instance the proportion of males and 
females. 

Random sampling ensures that every individual has an equal chance of being selected and therefore should be representative of the total 
population. This is based upon the assumption that, while individual events cannot be predicted with accuracy, aggregate events can. For 
instance, although we may not be able to predict an individual's academic performance, we can predict accurately the average academic 
performance of a group. 
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3.1.1 Confidence intervals 

A concern of every researcher is the confidence with which they can infer that their findings, based on a small subset of respondents, are an 
accurate reflection of the views of the broader population. A way to measure this is by calculating the confidence interval. Confidence intervals 
show the range of values around a sample or survey figure which can be expected to contain the true butunknown value forthe entire population 
on usually 95% of occasions. They are a measure of the sampling error (chance variation that occurs in sampling) and do not take account of 
differences caused by selection bias (caused by using unrepresentative samples). 

For example, a survey of 250 disaffected young people suggests that 50% of them have been involved in training programmes. The confidence 
interval calculation (see Appendix 1) demonstrates that we can be 95% certain that if we asked all the disaffected young people, the figure of 
young people involved in training programmes would lie within the range 44% to 56% (confidence interval of +/- 6%). 

3.1.2 Sample size 

The way in which you select your sample and the size of that sample is dependent on a number of factors, not least the scale and scope of the 
evaluation. Small-scale localised studies of a particular group of young people are unlikely to require sophisticated sampling techniques. 
However, you may wish to consider the following factors if you intend to undertake larger county or area-wide surveys: 

• Confidence interval required. In order to reach a specific level of confidence, a minimum sample size must be achieved. 

• Method(s) of data collection. Postal surveys, for example, typically yield a 20-30% response rate, although measures such as 
reminder mailshots can be used to increase this percentage. If a minimum of 250 responses is required, a sample of between 1,000 
and 1,500 will need to be selected. 

The availability of target audience and cost/ resource factors are also legitimate considerations when determining the appropriate sample size. 
3.2 Quantitative and Qualitative Approaches 

The relationship between theory and practice can easily be misconstrued, the former being seen as too removed from reality and the latter being 
criticised for ignoring results of current research. 
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ver the last decade, there has been a division between the quantitative and qualitative research traditions. Certain research processes tend to 
be associated with each side of the divide. Although there are constraints on what methods can be combined, there are many different forms of 
data collection and analysis that can be used in the evaluation process. Evaluation strategies often combine qualitative and quantitative methods 
in various ways and to varying degrees. 

Q ualitative research examines not the prevalence but the nature of the issue(s) under investigation. It explores how people think about things 
and what issues are important and relevant to them. Personal accounts, although subjective in nature, can provide in-depth and highly 
contextualised information about the subject under evaluation. Q ualitative approaches have the following main features: 



• an interest in meaning, perspectives and understandings; 

• an emphasis on 'process' rather than 'producf; 

• N ot considering issues in isolation but taking account of social context. 



Because the Connexions strategy is designed to respond ho listica My to young people's needs and responsibilities, qualitative research may seem 
to be particularly appropriate. Indeed, holistic assessments of young people's needs is an essential element of the Connexions strategy (DfEE, 
2000b). Its main aim is to provide assessment services that support young people by identifying and overcoming the full range of youth issues 
that can block successful progression. 

Q uantitative research is concerned more with objectivity and the precise measurement of the activities and events that are being evaluated. 
Q uantitative approaches have the following main features: 

• a search for causal relationships, the identification of specific circumstances that cause similar outcomes in different settings. 

• standardised research instruments designed to test or measure theories and hypotheses, producing numerical data for subsequent 
statistical analysis. 



Statistical data are most commonly analysed to produce: 



• Frequency: the number or percentage of times that the item occurs. 

• Mean: the arithmetical average of scores. 

• M edian: the point on the scale of measurement below which 50% of all scores fall 

• Mode: the most frequently occurring score in the data. 
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The Statistical Package for the Social Sciences (SPSS) is often used to carry out statistical analysis. However, spreadsheets and databases such 
as Excel and Access can be used for more basic procedures. Data analysis is a complex process and it is important therefore to invest in 
appropriate training to develop the right level of technical and statistical expertise. 



3.2.1 Identification and analysis of existing research 



ne of the first steps of any evaluation is to find out what research and information already exists on the chosen topic. However, it is important 
to be confident that this data is both relevant to the project, and reliable, before it is used. 



r For example: ^ 
If academic achievement in disaffected young people is under investigation, it may be possible to negotiate access to existing school 
records in order to get baseline information without needing to undertake further research. 



Searching for existing research and information before collecting new data helps to save time and resources, and to avoid 're-inventing wheels'. 
It can also provide contextual information in support of your rationale. Finally, existing data can be integrated into the project findings to draw 
additional conclusions and/or substantiate the work. 

3.3 Useful Research Methods 

The following sections detail a range of research methods that you may find useful in order to carry out your evaluation. 
3.3.1 Q uestionnaires (structured interviewing) 

Q uestionnaires can be used for both qualitative and quantitative research. When questionnaires are used in qualitative research, the questions 
tend to be 'open' in order to allow in-depth exploration of a subject by encouraging the respondent to provide detailed information in their own 
words. This enables the researcher to establish the context and the reasons behind the responses. pen questions require greater effort on the 
part of the respondent and the responses themselves may sometimes be difficult to interpret and summarise. 

When questionnaires are used in quantitative research, 'restricted or closed' questions are used. Here the respondent marks yes or no, gives 
short answers, or checks an item from a list of suggested responses. Closed questions are easier and less time-consuming to complete but do 
not provide the contextual detail provided by open questions. 
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Attitude scales are a type of closed question. When constructing questions of this type, the Likert Scale technique is often used. This technique 
assigns a scale value to each of the (most commonly, five) responses: for example, 1 = strongly disagree, 2 = disagree, 3 = undecided, 
4 = agree and 5 = strongly agree. Responses to this type of question are considerably enhanced if the respondent is given the opportunity to 
elaborate on the reasons for their answer. Validity is improved because it is more difficult for the respondent to conceal their true attitudes and 
express socially acceptable opinions. 

Many questionnaires include both closed and open questions. Each has its merits and limitations. A balanced approach must be taken, informed 
by the type of information and analysis required. 

Q uestionnaires can be administered by a researcher either face-to-face or over the telephone (structured interview), or they can be designed for 
self-completion by the respondent. Personally administered questionnaires give the researcher the opportunity to build rapport, explain the 
purpose of the study and clarify any items that are not clear. However, in order to maintain objectivity, the researcher should take care not to 
influence or prompt particular responses through the additional information they may provide. Self-completion questionnaires can preserve 
objectivity and have benefits in terms of cost, time-scale and coverage. 

Constructing your questionnaire 

Having established that a questionnaire is the most appropriate research method, there are a number of aspects that should be considered 
before the questionnaire is constructed: 

• What do you want to find out? 

• Who will you find it out from - target audience? 

• How will you gain their involvement - what's in it for them? 

• How will the questionnaire be administered - face-to-face, over the telephone or self-completion? 

• How will you analyse the information obtained? 

• How will you present your findings? 
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Characteristics of a good questionnaire 

• It seeks out information that cannot be obtained from other sources. 

• It deals with a topic that is relevant and important to the respondent. Information is provided about the purpose of the research, the 
sponsor, the sample-selection procedure and how the results will be used. 

• It is well organised, with the items presented in a logical order from general to specific responses; clear directions are provided on 
how to answer the questions. 

• It is as succinct as possible, yet long enough to obtain all the essential information. 

• It has been piloted and appropriately revised. 

• It is designed so that the researcher can easily analyse and interpret the findings. 

• The questions are as objective as possible and do not prompt or lead the respondent. 

• The use of descriptive words with meanings that are open to interpretation (e.g. frequently, occasionally and rarely) are, where 
possible, avoided and unambiguous words and phrases (e.g. 3 times per week) used instead. 

• The 'Plain English' principle is applied, ensuring that questions are worded clearly and simply and only one piece of information is 
requested at a time. 

• The pre-coded questions provide a suitable list of possible answers as well as an 'other' category 

• The questions are phrased in a neutral way to avoid 'acquiescence bias' that is, the tendency for people to tacitly agree with a given 
statement. 

• The validity and reliability of questions have been verified. 

Piloting 

It is good practice to 'pilot' or test a questionnaire with a small initial sample of respondents before administering it to the target group. Piloting 
can establish whether: 

• the topic is relevant to the respondent; 

• the wording of questions, definitions and descriptions is clear; 

• the list of alternative responses to pre-coded questions is adequate; 

• the questionnaire flows logically; 

• all the pertinent issues are covered. 
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Failure to pilot a questionnaire can have serious consequences for the evaluation study: 

• Respondents may misinterpret questions and so give incorrect or incomplete responses and conclusions. 

• The questionnaire is less likely to be valid or reliable. 

• In the worst-case scenario, the questionnaire may have to be re-administered, with the risk of alienating the target group. 

• Re-administering the questionnaire also has cost and time implications. 

3.3.2 Unstructured and semi-structured interviewing 

Unstructured and semi-structured interviews facilitate a conversation between the interviewer and the respondent. This interaction takes place 
within a previously designed framework of themes and topics related to the subject of the study. With a skilful interviewer, this method can be 
superior to any other method of data collection. 

Unstructured and semi-structured interviews are used when there is a need to: 

• collect detailed individual accounts or histories; 

• collect in-depth information about personal motivation or behaviour; 

• understand complex circumstances; 

• discuss issues that might be of a sensitive or threatening nature. 

Types of interview, include: 

• Semi-structured interviews - sections of pre-defined questions to gain specific information about key issues are used in conjunction 
with unstructured exploratory discussions. 

• Investigative interviews - all topics are covered with every respondent (possibly even in the same order), but with varying levels of 
depth or details per topic, depending on the respondent being interviewed. 

• In-depth interviews - there will be certain key questions (probably few in number) that need to be answered but how these questions 
are addressed will depend on what is said by the participant. Lines of questioning will be continued until a clear understanding of 
the subject in question has been gained. 

• rganic interviews - these have a more conversational, informal structure. Ideas and topics for conversation are generated by the 
interchange between the interviewer and the respondent. There may not be any advance agenda on the part of the interviewer. 

o 
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The most accurate way to document an interview is to tape-record the discussion with the respondent's consent. The transcripts generated by 
the interviews then need to be analysed. Usually thematic/content analysis is best fitted for this type of information. There are computer software 
programmes that cluster the emerging patterns or common themes; for example, N4C lassie (formerly known as NUD*IST: Non-numerical 
Unstructured Data - Indexing, Searching and Theorising), although this can also be done manually 

Advantages of such interviews include: 

• A good interviewer can establish rapport with the respondent easily, forming a trusting and secure relationship within which 
confidential information can be relayed. 

• An interviewer can explain more easily and clearly the purpose of the investigation and can clarify questions, minimising the 
possibility of misinterpretation. 

• A fuller, more in-depth understanding of the issues and the meaning behind the responses can be gained. 
Disadvantages include: 

• Arranging and conducting interviews is a time-consuming procedure. 

• The presence of the interviewer may bias the responses. It is important to take into consideration ethnicity, gender and possibly other 
characteristics of the interviewer in relation to the interviewee. 

• The responses and subsequent interpretations are subjective. Thus, it is important to develop a framework that will guide the 
interviewer to raise issues and obtain information on specified themes/topics. 
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3.3.3 Focus group discussions 

Focus group discussions provide opportunities to study interchanges between participants. Properly and appropriately conducted, they can be 
particularly valuable in: 

• highlighting shared or common experiences; 

• identifying different or polarised views; 

• acting as a trigger for wide-ranging debate and stimulation of ideas. 

The focus group should be moderated by someone who is knowledgeable about the subject and skilled at group facilitation. The moderator 
should take their steer from a thematic topic guide' similarto thatused in an unstructured interview but should be careful notto prompt or unduly 
lead the participants. A skilled moderator will ensure that everyone is given the opportunity to contribute their views and not allow one or two 
individuals to dominate the discussion. It is good practice for the moderator to introduce the group by providing a brief overview of the project 
and asking group members to introduce themselves with information that is relevant to the subject matter. G roups can be re-convened on more 
than one occasion, which is useful if there is a need to absorb and reflect upon information presented on previous occasions. 

Focus group discussions are often used at the initial stages of a research or evaluation project. They can also be used to explore the range and 
diversity of views on an issue; to consult members of the population about an issue or a proposed plan for action; to enable people in similar 
circumstances to discuss their common experiences and needs; test reactions to promotional or publicity materials. 

The most accurate way to document the outcomes from a focus group is to tape-record the discussion. However, transcribing and analysing this 
information is very time-consuming and plenty of time should be allocated to this stage in the process. If you decide to adopt this approach, it 
is essential to gain the consent of the participants in advance and to ensure that they are comfortable with this arrangement. Some people do 
not feel at ease being recorded, perhaps due to shyness or past negative experiences. In this instance, someone other than the moderator should 
be present to take detailed notes of the discussion. 

The transcripts can be analysed in a similar way to an unstructured interview, using thematic/ content analysis. Particular note should be taken 
of points of consensus and disparity. 
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When convening a focus group there are a number of important factors that should be considered: 

• Group size. The optimum size for a group discussion is 7-8 people. This allows everybody in the group to participate and also 
provides a sufficient base for a broad discussion. However, it may be advisable to invite up to 15 participants per group, as it is 
likely that not everyone will attend. 

• Group composition. This depends on the purpose of the discussion. In some circumstances it is appropriate to get a mix of 
participants with different characteristics (e.g. diverse interests, different age groups), at other times it is better to ensure participants 
have some characteristics in common (e.g. occupational or study subject area). 

• Timing. The group should be held on a day and at a time that is convenient to the majority of participants in order to ensure a high 
turn-out. 

• Location. A suitable venue should be booked well in advance. It should be accessible to those with mobility problems. Try to ensure 
that the room is self-contained and in a quiet location, particularly if the intention is to tape-record the discussion. The room should 
be close to toilet facilities. 

• Health and safety. The health, safety and personal security of the moderator and the participants is of the utmost importance. 
Colleagues, managers and key holders should be informed of the date, time and venue of the group. Security arrangements should 
also be verified. 

• Resources and equipment. If you intend to tape-record the discussion ensure that the room has a plug socket, tape recorder and 
extension lead if required. If battery-operated recording equipment is to be used, ensure that a spare set of batteries is available. A 
flip chart is a useful aid to record the main points. If the participants will be required to write anything down, ensure that there are 
enough pens. 

• Incentives. Providing participants with an incentive for attending. A book voucher, for example, encourages attendance. It is also a 
good idea to supply refreshments such as tea, coffee and water. 
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Advantages of focus group discussions include: 

• They allow participants to discuss their common experiences and bring out 'difficult' views in a shared environment. 

• They provide a forum for expression to those with poor reading and writing skills. 

• Issues and ideas are generated within the group that may not have been raised by individuals. 

Disadvantages include: 

• Group discussions need expert facilitation and a recording system. 

• The discussion can be dominated by more confident and articulate members of the group. 

• Some group members may feel inhibited when discussing issues of a sensitive or threatening nature, especially if they are being 
recorded. 

• Although the information can be very 'rich', it is time-consuming to analyse. 
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Chapter 4: Reporting and Dissemination 



4.1 Reporting 

nee the fieldwork and analysis is complete, the findings need to be presented in a clear and coherent report that is suitable for the audience(s), 
who may include practitioners, managers and policy makers. A typical report structure may contain the following sections or chapters: 

• Abstract - a brief paragraph that sums up the essence of the evaluation. 

• Executive Summary - a short synopsis that summaries the main findings, conclusions and recommendations, usually in bullet- 
point form. 

• Introduction - this section makes explicit the aims, objectives and rationale for the evaluation. It often contains a summary of the 
existing research on the topic area in order to contextualise the project. 

• Methodology - this section sets out the methods used for data collection and the reasons for their adoption. 

• Findings - one or more sections, organised in themes, to summarise the findings and the evidence to support them. 

• Conclusion - the final section draws the main findings together and clearly sets out relevant recommendations for policy, practice 
and further research. 

• Appendices - sample questionnaires and topic guides from focus groups and interviews can be included as appendices. 

4.2 Communication and Dissemination 

Presenting the evaluation findings in a clear, concise, accessible format and ensuring that they are effectively disseminated to key stakeholders 
and other interested parties is essential for a number of reasons. 

An effective dissemination strategy helps to ensure that the evaluation process remains open and transparent, and prevents cynicism about the 
process itself and how the findings are used. Research and evaluation activities rely on the co-operation of key stakeholders, who may become 
reticent about their continued participation if they are not made aware of the benefits of, and the rationale for, their participation. It is essential, 
therefore, to identify ways of sharing findings and feedback to ensure that individuals and organisations benefit from the work. 
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Broad dissemination of research findings can create a 'multiplier effect' in that the project outcomes can be utilised at different levels inside and 
outside the organisation. This: 

• ensures that good and interesting practice is shared; 

• prevents others utilising scarce resources to replicate existing research; 

• provides practitioners, managers and policy makers with the evidence needed to justify the approach taken to specific tasks, 
strategies and policies. 

Within the Connexions Service, findings can be disseminated via partnership activities with individuals and agencies. M ethods of dissemination 
may include: 

• A newsletter - both in hard copy and through ICT systems. 

• A website with indexing and referencing; 

• Cross-evaluation events; 

• National, regional and local conferences. 
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Chapter 5: Making it Work 



Whether you are undertaking a small-scale local evaluation or a large-scale regional study, the same basic principles apply to 'making it work'. 

5.1 Preparation 

The top ten questions to ask yourself when preparing to conduct an evaluation project are as follows: 

1 What do I want to evaluate and do I have a clear rationale for doing it? 

2 What research or baseline information already exists? 

3 Is the timescale for collecting, analysing and presenting information realistic and achievable? 

4 Do I have permission to undertake the evaluation and the commitment a nd supportof key stakeholders? 

5 What are the key indicators of success? 

6 How do I intend to evaluate the outcome(s) and compare them to the initial aims and objectives of the project? 

7 Can I clearly identify the deliverable outcomes from the project? 

8 Have I piloted the research instrument(s) before embarking on the main evaluation activity? 

9 Have I dealt with and/or assessed the reliability and validity of the evaluation procedures? 

10 Have I developed effective communication protocols and/or channels for disseminating the results appropriately and widely? 
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5.2 Reflection 

In order to assess the relative success of the evaluation, the project manager should reflect on the following: 

1 Aims and objectives. Were all the aims and objectives met? If not, why not? 

2 Methods. What was the value of using the chosen methods? Were they the most useful and appropriate for the task? 

3 Project management. What were the main lessons learned? What, if anything, should be changed, developed and retained if this 
project was to be repeated? 

4 Findings. Are the results and/or outcomes valid and reliable? Do they 'add value' to the field of study/practice? 
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Project 
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Admin 
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Initial Steering Group meeting to agree the parameters of the project 
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Evaluation Team Meeting 
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Develop postal survey for distribution to ** partner organisations 
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Develop fieldwork visit outline proformas and written report specification 
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Evaluation Team Meeting 
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Phase 3: Fieldwork 


Visits and/or telephone interviews 
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database for the mailing 


























Fieldwork visit pack to be produced 


























Postal survey including sae/questionnaire and telephone work with an 
approximate two week turn around for responses 


























Team meeting to review progress and finalise plans for the fieldwork activities 



























Visits and/or telephone contacts with Contract Manager, Strategic Managers, 
Co-ordinators 


























** one-to-one interviews conducted face-to-face 


























To review the evidence from the completed fieldwork 















































































Reader to quality assure final report before forwarding to the relevant parties 
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Confidence Interval Table 



' Number of respondents 


Confidence interval ^ 


50 


±14 


70 


±12 


100 


±10 


150 


±8 


270 


±6 


600 


±4 


1070 


±3 


2400 


±2 


^ 9866 





Confidence interval calculation 



Square rootof: (response rate x [100 -response rate] ) multiplied by 1.96 

( Sample size ) 

E.g. 74% of respondents agreed with a specified statement from a sample of 1,250. 

Response rate = 74 multiplied by (100 - response rate 74) 26 = 1,924 

1,924 divided by sample = 1,250 = 1.54 

Square rootof 1.54 = 1.24097 

1.24097 multiplied by 1.96 = 2.43 

Confidence Interval = ±2.43% 

In order to calculate the maximum confidence interval of any given sample, use a response rate of 50% 
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